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Remarks 

Reconsideration and allowance of this application are respectfully 
requested in view of the above amendments and the following remarks. 

By this Amendment, the specification has been replaced with a substitute 
specification; two drawings are added (for reasons explained below); the abstract 
of the disclosure has been replaced; claims 1, 2, 4, 10, 1 1, 13, 18, 20, and 21 have 
been amended, and new claims 22 — 36 have been added. Upon entry of this 
Amendment, claims 1-4, 9-14 and 18-36 will be pending in this application. 

As a preliminary matter, this application is a continuation of the parent 
case 09/906,288, but not a divisional thereof. Applicant's designation of this 
application as a divisional was an inadvertent error. 

In response to the Examiner's request, applicant has submitted a new 
abstract herewith- Approval of this Abstract is respectfully requested. 

Specification and Drawings 

Applicant also submits herewith a substitute specification per 37 CFR 

1.125. This substitute specification includes no new matter. A marked up version 

of the substitute specification is included. 

The specification has been amended to deal with issues raised by the" 

Examiner as well as to correct minor typographical errors found by applicant. In 

addition, the Examiner requested that the embedded hyperlinks (in original" 

paragraphs H0007 and ^0073) be deleted. Those paragraphs (now 1(0008 and 

H0077, respectively) have been amended as follows: 



Paragraph 0007 as filed 


is now Paragraph 0008 


DNS Lookup (Seconds): The time it 
takes for the browser to turn the text 
based hostname (e.g., www.yahoo.com) 
into an IP address (207.221.189.100). 


DNS (Domain Name System) Lookup 
(Seconds): The time it takes for the 
browser to turn the text based hostname 
(e.g., as shown in FIG, 6) into an IP 
address (207.221.189.100). 
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Paragraph 0073 as filed 


is now Paragraph 0077 


FIG. 3 is * * * Specifically, in FIG. 3, 

the displayed statistics relate to a web 

site having the URL: 

"http ://r ocky .adquest3 d.com: 8 1 /superm 

ainxfin?brd=9000". 


FIG- 3 is * * * Specifically, in FIG. 3, 
the displayed statistics relate to a web 
site having the URL shown in FI<3. 3A. 



Approval and entry of the substitute specification is respectfully requested. 

In order to deal with the required amendments to the specification, two new 
drawings have been added. Specifically, new Figure 6 contains the hostname 
previously listed in paragraph H0007 and new Figure 3 A contains the URL " 
previously listed in paragraph TJ0073. No new matter is added by these two 
drawings and their entry is respectfully requested. 

The Claims 

The Claims have been amended to deal with the issues raised by the 
Examiner and to clarify certain aspects of the invention. 

Claim 1 has been amended to recite that "the processing module is further 
programmed to alert a user about a potential network performance problem based, 
at least in part, on whether the predetermined relationship exists between the first 
and second variance statistics." Claim 10 has been amended to recite "alerting a 
user about potential network performance problems based* at least in part, on 
whether the predetermined relationship exists between the first and second 
variance statistics." Support for these claim amendments is found in the 
application as filed, e.g., at TJ0081 (now 10085). [Note that these and other" 
references to the specification as provided only are examples, and are not irk any 
way meant to be limiting.] 

Claim 20 has been amended to recite ''determining whether a potential 
problem exists with the network's performance based, at least in part, on a 
comparison of the first and second variance statistics; and generating an alarm 
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when it is determined that a problem exists with the network's performance." 
Claim 21 has been amended to recite that "the processor is further programmed to 
determine whether a potential problem exists with the network's performance 
based, at least in part, on a comparison of the first and second variance statistics; 
and to generate an alarm when it is determined that a problem exists with tixe 
network's performance." Support for these claim amendments is found in the 
application as filed, e.g., at 1J0055 (now Tf0057). 

Support for new claims 22 and 24 is found, e.g., at original 1|0053 
(substitute H0055). : * 

Support for new claims 23 and 25 is found, e.g., at original 1J0064 
(substitute TJ0068). * 

Support for new claim 26 is found, e.g., at original T[008 1 (substitute 
H0085). 

Support for new claim 27 is found, e.g., at original 1HI0032, 0040 (substitute 
W>033 and 0043) : : 

New claims 28-34 correspond to original claims 5-8 and 15-17, 
respectively. 

Support for new claim 35 is found, e.g., at original ^[0005-0026, and 0041 
(substitute ffl[0006-0027 and 0044). 

Support for new claim 36 is found, e.g., at original 1(0045 (substitute 
^0048). 

Prior Art Rejection 

The Examiner rejected claims 1-4, 9-14 and 18-21 under 35 USC §102 as 

being anticipated by "Variance of Aggregated Web Traffic" (Morris). Hie 

grounds for this rejection are respectfully traversed in view of the above 

amendments and the following remarks. 

Morris is an academic paper discussing statistical models of web traffic and 

their relationship to real traffic. See, e.g., Abstract Morris, however, lacks any 
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teaching whatsoever of the claimed alert mechanism. Morris does not provide any 
real-world application of his results, he merely 4< present[s] evidence that 
bandwidth variance changes linearly with the mean." Pg. 365, § VII. In fact, 
Morris does not teach a dynamic (ongoing) monitoring of a network. Instead, 
Morris monitored network traffic in order to evaluate a theoretical model. 

Claims 1, 10, 20, 21, 26 and 27 are independent. Each of the claims 
includes at least one feature that is not taught or in any way suggested by Morris 
(or any of the other prior art of record). 

As noted above, claim 1 has been amended to recite that "the processing 
module is further programmed to alert a user about a potential network 
performance problem based, at least in part, on whether the predetermined 
relationship exists between the first and second variance statistics." There is 
nothing in Morris to teach or in any way suggest such an alerting feature. 

Similarly, claim 10 has been amended to recite "alerting a user about 
potential network performance problems based, at least in part, on whether the 
predetermined relationship exists between the first and second variance statistics." 
Again, there is nothing in Morris to teach or in any way suggest such an alerting 
feature. 

Claim 70 has been amended to recite: "determining whether a potential 
problem exists with the network's performance based, at least in part, on a 
comparison of the first and second variance statistics; and generating an alarm 
when it is determined that a problem exists with the network's performance." 
MoTris is silent as to any "determining whether any problem exists with a 
networks performance/ 5 As noted above, Morris is an academic paper in which he 
describes using real data to confirm a theoretical model. Thus, Morris does" not 
teach or in any way suggest at least this aspect of claim 20. 

Similarly, claim 21 has been amended to recite that "the processor is, further 
programmed to determine whether a potential problem exists with the network's 
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performance based, at least in part, on a comparison of the first and second 
variance statistics; and to generate an alarm when it is determined that a problem 
exists with the network's performance." For reasons similar to those for claim 20, 
Morris lacks any teaching or suggestion of at least this element of claim 21. 

New independent claim 26 recites a method that includes "alerting a user 
about network performance that is outside acceptable tolerance or control limits. 5 ' 
Morris has no such teaching or suggestion. ' 

New independent claim 27 recites a method that includes "automatically 
calculating variance statistics * * *; and using a history of performance data 
statistics to alert a user about performance of network services that are outside 
acceptable tolerance or control limits. 59 Again, Morris lacks any teaching or 
suggestion of the claimed alerting. 

The other claims all depend (directly or otherwise) from these claims. 

Thus, as shown, there is at least one element of each claim that is not taught 
or in any way suggested by Morris. Accordingly, withdrawal of this rejection 
under § 102 is respectfully requested. 
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CONCLUSION 

In view of the foregoing amendments and remarks, allowance of this case is 
earnestly solicited. 1 

The Examiner is invited to telephone the undersigned should he believe that 
a telephone conference would resolve any outstanding issues in this application. 



CHARGE STATEMENT : The Commissioner is hereby authorized to charge any fee specifically autborized hereafter, 
or any missing or insufficient fee(s) filed, or asserted to be filed, or which should have been filed herewith of 
concerning any paper filed hereafter, and which may be required under Rules 16-18 (tmssing or insufficiencies only) 
now or hereafter relative to this application and the resulting Official document under Rule 20, or credit any "_ 
overpayment, to our Account/Order Nos. shown in the heading hereof for which purpose a duplicate copy of this sheet 
is attached. 




Respectfully, sjubmitte 



By: 



Davidson Berquist Jackson & Gowdey LLP 
4300 Wilson Boulevard; Suite 700 
Arlington, VA 22203 
Main: (703) 894-6400 
FAX: (703) 894-6430 





Brian Siritzl 
Registration No.: 37,497 
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' System and method For Providing Composite Variance 
Analysis For Network Operation 

Related Aftlic attons 

[00011 This application claims priority to and is a contin natinn of U.S. 

Patent Apphcaiion No. 09/906.288. filed July 16. 2001. now U.S. Patent No. 
6.708.137. which is incorporated herein in its entirety. 

Technic ax* field 

rOQ021 T he present invention relates to performance of information 
networks,. In particular the present invention relates to statistical measurements of 
performance characteristics of an information network. 

Baokgrotjtto 

100031. • Internet web sites continue to become more sophisticated and offer a^'- : _^^^^^^^^^^^ -j 
wider variety of media for a user to access. With this trend, users have become . \ •■ • 

V"" . V - * " J Dpfl«t*di tablet " . 1 

more demanding of quick, high quality Jgjejneie^ ' ' /:■• ~ — 

to keep -up with user's demands, it has became increasingly important for the " . 

provicfere of ^ternrt 



\ F6tttt3jttedi Tabs: Not at Q* : ] 



Delated: 



performance issues to both avoid degraded performance and provide unproved 
perforrnarice. 

f 00041 Given this, systems have been developed for measuring relevant 

network parameters to evaluate network perfomiance and help troubleshbot 
network issijes which might degrade network performance. Generally, such 
systems utilize computer servers deployed on a network of interest to measure 
network performance parameters. .Such computer servers are generally referred to % 
as data collection agents. (DCAs). .A DCA generally connects to a device in the s 
network about which a measurement is desired and takes one or more 
measurements of one or more predetermined metrics The DCA then, typically 
stores the results of the measurement either locally or in a remote database.. The^ 
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stored measurements can Jhen, be called up and reviewed by a user who accesses 



_Such systems can typically measure metrics related to either 



fOOOST 



Universal Resource Locator (URL) objects (such as a web page located on a 
server on the network) or streaming media objects. URL objects and streaming . 
media objects are collectively referred to herein as network services. With respect 

to URL objects, such metrics can include, but are not limited to- v . ^ 

\000€\ End-to-End Time (Seconds): The time taken from the moment a 

user clicks on a link to the instant the page is fully downloaded and displayed. It _ 
encompasses the collection of all objects making up a page including, but not 
limited to, third party content on off-site servers, graphics* frames, and 
redirections. 

[0007] .Throughput (KB/Sec): The amount of data streamed back to the ! 

user and how. long it took (in kilobytes per. second). The calculation is based on 
adding.all data segments returned (for example, but not limited to, the body of 
; HTML Pflvper Text Markup Language], d ocuments and images) and dividing that ", 
by the total time it took to return that part of the data. It is to be understood that 
browsers and servers are requesting objects in parallel so throughput does not 
represent the limit of a Web server in returning data. 

100081 D NS (Domain Name System) L ookup (Seconds): The time it takes 

for the browser to turn the text based hostname (e.g., gg shown t in FIG -,6) into an . 
IP address (207^21.189.100). 

r00O91 Connect Time (Seconds): The time h takes to set up a network 

connection from the end-user's browser to a web site. A web page is transferred 
over this connection and many are setup for each page. 

rooioi Request Time (Seconds): The time it takes to send a request from a 

user's browser to a server. This is a relevant amount of time if you are submitting a 
| large form (e. gwLAS^. 5 ^^^^^-^- 9S^^S!LS^S!^^S£S& ISifcJSL :.— 
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attachment to a message on a discussion board). It reflects the ability- of a server to 
accept data. * 
room . . R esponse Time (Seconds): The time it takes for a, server to respond 
with content to the browser. Preferably, this measurement is talpen by waiting until 
the first byte of content is returned to the browser. 
| ] 00121 T eardown Time (Seconds): The time it takes for the browser and] . 
" server to disconnect fixjm each other. , 
| r0013T . Download Tiwe (Seconds): Hie time for the page download from 
the start of the first object to the end of the last object 

f0014l H ie unit in parenthesis following the name of the metric is the unit 

in which the measurement is generally taken and recorded. 

| f00151 With respect to Streaming media objects, such metrics include, but 

are not limited to: ... 

rofllpl ;DNS Lookup Time (seconds): This metric is generally the same as 

the DNS lookup time for URL type objects. 

room Q uantity of Data Received (bytes or bits): The absolute amount of 

data gathered by the DCA if a stream had been rendered. 

rOQlSl : P acket Loss (muilber): Hie number of packets that are not received 

by the media monitor, 
j [Q0191 Percent Packet Loss (number): The percentage of total packets, that 
. are not received by the media monitor. - 

r 00201 Packets Received (number): The total number of packets received 

by the media monitor. 

[002H Packets Late (number): The number of packets received too late to 

functionally render. * 

f00221 . P ackets Resend Reque8te6^(numb^ 

have been requested to be resent This metric preferably applies, to 

m^mm^^^?^^ ; , L~ ^ 
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[00231 P ackets Recovered (number^Tbe number of packets for which ■ J ■ ~ — '' " ■ :.r 

some type of corrective action is taken. "Corrective action 0 typically means • ... % • ■*•■:. \ 

requesting that the missing or broken packets be resent This metric preferably 
applies to^£^MBP^™sgg^ag. _ _ 



^\ Delet ed! RrtP^to-KJM. 
A Formatted: ftxtfc NdI 

[0024] Packets R^en^^iimber)^£AlsoJknowi as jga^etsjesmd Received) jj^. ^ — -r s • % . . . 

the number of packets asked for again (the packets resend requested metric) and ; . i « ; v 

were received. This metric preferably applies to ^ALMEPI/V™ streams. „^-^ — 

f00251 > Packets Received Normally (number): The number of packets •„."."■ . 

received by the media monitor from the streaming media server without incident . ; 

| r00261 Current Bandwidtl^y^sw^^^ si:^ 



Font: Not Bofei 



received measured over a relatively small time frame. . . 

f00271 ■ C lio Bandwjdtl^Q^e^secg^ ^ Not ^ 

measured over the length of the entire stream or over a relatively long „ • 



[QQ^T . R esults of the above measurements can be used to help determine 
whether network services operating up to standard.- In the context oftheJfatg?SgL_ 
results of the above URL object measurements can, for instance, indicate whether 
a web page is downloading consistently, at a higji enough speed, or completely. 
The results of measurements of the above streaming media parameters can help 
determine the same information with respect to a streaming media object 
r00291 H owever, while important diagnostic information can be collected 

about the current status of a particuJtar web page or streaming media service by 
making individual or random measurements of one or more of the above noted 
network performance.metrics, it can be difficult to use this testing method to fully 
diagnose performance. For example, using such techniques it can be difficult to . 
determine the performance of a network over time or during certain times of the * 
day, days of the week, or parts of the year. Thus, it can be difficult to detect, and 
predict, cycles in network operation, such as. if a network operates more and less 
rapidly on a periodic basis. Such information could be useful in determining how 
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oUiei network parameters "such as network traffic, load, which likely -varies oyer -a-' 
day, week or year period, effect! performance of network services. 
JOQ30T - . .W ithout such information, individual measarrements may be . ! " 
.r^sleading. For example, an tn^atufkctory results of such measurement may be ; . 
caused by high, or low network traffic load, rather than a specific problem with a . . 
. network device. Also, using the above de$cribed standard techniques, it can be 
difficult to provide any .type of predictive event correlation. For example, what, if . 
any is the. effect of degradation of. DNS lookup time on overall network service 
performance during specific time periods? Such predictive information can help 
providers of network services to set: appropriate expectations of network 
performance for customers of such providers. Additionally, such predictive 
information can facilitate troubleshooting of root causes relating to network, ■ 

application and third party content (e.gp. banner ads on ft web site), issues. , ; 

f003ll . F urther, in order to determine whether a particular network service is 
operating appropriately using the above described methods, a user must initiate 
measurement of one or more network performance metrics, retrieve and then 
analyze the result That is, there is no way for a system that does no more than 
take measurements of network performance metrics to notify a user if a network is ■ 
not operating correctly because there is no baseline or other reference available to 
the system to jnake such a determination, 
j r0Q321 W hat is needed is a system for measuring network performance 

metrics which allows a user to take into account network conditions, such a traffic 
load, when .analyzing the measurement Also, the system should allow a user to be 
able to make predictions about network performance at a given time. Additionally, 
such a system should be automated and -should be able to analyze and present 
measurement results in a manner which is meaningful and straightforward to 
interpret 
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' . SUMMARY OX TiSE INVENTION 
[00331 . .. A system. and method in accordance with tfcepresent invention .■ 

collets measuremeats.of ^network performance metrics and au^ 

calculates and provides composite variance analysis of such metrics. The system 

and method can then use history of performance data statistics to alert a user about 

performance of network services that are outside acceptable tolerance or control 

limits. The technique exposes- subtle deviation from accepted measurement 

tolerance that can, in turn, be categorized in relation to control limits based on 

defined standard deviation thresholds. 

[00341 , A system in accordance with the present invention includes at least s 
one DCA located on a network, a processing module interconnected with the 
DCA, and, preferably, a comparison module interconnected with the processing . , 
module. The DCA collects at least a first plurality of measurements of a single 
network parameter and at least a first set of measurements including at least a 
single measurement of the single network parameter. Each of the first plurality of 
measurements is taken at a different time. The processing module calculates at 
least a first variance statistic, such as an average value, and a second variance 
statistic. The first variance statistic relates to the first plurality of measurements 
and the second variance statistic relates to the first set of measurements. The 
comparison module compares the first variance statistic with at least the second 
variance statistic to determine if a predetermined relationship exists between the 
first variance statistic and. the second variance statistic. For example, the variance 
statistics could be averages of the group and first.set of measurements. The 
comparison module could determine if the average of the first set of measurements 
is within a predetermined multiple of standard deviations from the average of the 
group of measurements. Preferably, the system also includes a screen display for 
displaying at least the first and second variance statistics and the results of the 
comparison thereof. 
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. 100351 A method in accordance with the present invention includes /. ; v . -r \ •> - - 

collecting at a f^plura^^ v ;7;:v • : x \\ *. 

mea^ement taken at a different titfe/^ *.*.;.• 
collected including at least a single mek^ement of the single network paramet».-\ . .-• :* ; 

Then a jHrst variance statistic associated with the first plurality of measurements .* . -: ; ' :i • v,;T \;\ n '.ri i , 
add at least a second variance statistic associated with the first set of 'I .\ 

measurements are calculated. The first variance statistic ia then compared with at t . ■ • 
leastthe second variance statistic to determine if a predeterjooiried relationship a ^ z '. :\ ..' 
exists the two variance statistics, 

Brief Description Of The drawings . : , - .v.. . 

100361 FIG. 1 is a block diagram illustrating a preferred embodiment of a 

method of the present invention f ur providing composite variance analysis for . • * • ■ i\- 

network operations in accordance ^yith the present invention. 
| r0037T FIG. 2 is a block diagram showing a system for providing composite ■" 

variance analysis for network operations in accordance with the present invention- . • 

J r00381 H G. 3 is a rer>roduction of a preferred embodiment of a screen 

display rendered by a page rendering module of a system in accordance with the 

present invention. 

10039]. gIG. 3 A is a URL of a web site whose statistics axe displayed in . ! . 

FIG. 3. 

rQ0401 F IG. 4 is a reproduction of the screen display reproduced in FIG. 3 ■ _ . 

showing a different portion of the screen display. ~ 

| [00411 FIG. 5 is a reproduction of the screen dispkyreproo^c^inFIG. 3 _ 

showing a different portion of the screen display. 
| [0042] F?Q, 0 is a URL of a hostname, - 

Detailed DESOflPTiON. r 
J [Q043] A system and method in accordance with the present invention 



FbrTnotbCdJTafe!Hotat QT ' / j 
Formatted; Bullets and-Numtemg^ 

collects measurements of network performance metrics and automatically 
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• /calculated 

and metodcan Aen useJiistoiy of performance data-statistics to alert a user about / 
*.i perfGfmaiK?e,of netwott services that are outsidV acceptable tolerance or control: 

lirnits/Thai is, a systemaiid method in accordance with the present invention 
• collects raw. data including a set of periodic irteasurements of at least a single 
network performance metric 'such gs, without, limitation, end-to-end time or 
throughput ofcat least a Single network service. Ciamposite variance analysis is' 
then completed on this .set of measurements; The results of this analysis are 
preferably values such as the average, mean, median, nSfcumum, maximum and 
standard deviation (referred to collectively herein as variance statistics) of the 
group of periodic measureanents of the singjemetric. The d 
analysis can be completed, with respect to any network performance metric or 
group of such metrics. Further, a set of periodic measurements for a single metric 
can be accumu^acted qver.any period of time. Accordingly, the results of the 
composite variance, analysis can advantageously be. used to determine how the 
. performance of a given, network service with respect Jo any desired, performance 
metric or groupof metrics varies over any amount of time, this also allows a user 
to advantageously determine whether performance of a network service at any 
particular time is. outside Qf acceptable limits. 

| ro0441 FIG. i is a block diagram illustrating a method 100 of providing 

composite variance analysis of network performance. In step 1 10, network 
performance date^^colle^ 
preferably inclu^^ 

performance metrics including, but not limited to,, those enumerated in the 
Background section with respect to both URL services and streaming media 
services such as DNS lookup time or packets lost Preferably, in step 110, method 
100 measures arid stores at least one network performance metric corresponding to . 
. : , a URL object or streaming media object on a continuous basis. With respect to 
ttRL objects, such metric can include, but is not limited to: throughput, DNS' 
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' lookup time,. comect". time, request time, server response time, teardown time, 
.. dowidoad time> aiuVor ' t . : 

such a metric can mclnde, but ls not limiM to: packets lost, packets reed 
bytes received, jitter, .percent packets lost, DNS lookup time, buffer time average v . v . , 
baadwicfth, and/cir first stafc time. Preferably -each.of the above listed metrics is . 
collected and stored in step 1 10 oh' a continuous basis. It is also considered to 
collect each metric, or a subset of the metrics, at only predetermined times. T m 
| 100451 . A q^tttibnallyJin stepi 110, method 100 preferably takes a ' . 

measurement of each of the above listed metrics approximately once per minute to " : ' 

take a total of approximately 60 measurements per hour of each metric on a ."■-.* 
continuous baisis. However, it is wito 

collect measurements pf the metrics at any . other interval of time. As discussed in ^ 
detail below, this information" can be stored in a database or other type of data . " : . 

storage configuration. '• . * . .. 

| f0M<fl P referably^ m step 110, method 100 collects error data relating to 
measurements made of network services. More preferably, step 110 collects errors 
referred to as access eiiois, service- errors and content errors. An access error 
■ includes an error that prevents a DC A fern startmg the download process jfor a , . 

given URL. A service error includes aDCA's failure to load the first object on the 
URL's page. Service errors can* occur, for example, when the cxmjaguration for a 
monitored URL is an "improperly forfaited or when the site is . 
content error includes an error that is encountered when downloading a component 
object for # URL. . .." .... * . " \. 

| f0Q47T JDq step 120, a user requests a report of collected network 

performance data and a composite variance analysis of such data. In makin g such . : • . . 

a request, the user preferably ineltidfes information identifying the URL of the site, 
or the streaming medjai service to be rneasur^ and a time range over . \ 

measurements £re desired, preferably in the form of a date range or single date . - • ■ . , ■ . . 
with a time range.' * ' '. ■• : 
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J00481 . A fter retrieving the raw network performance data corresponding to 
theUKL and time range' Of the .use* request,, in step 130, the retrieved raw network. 

. performance data^rej^^dtog^^^ 

variance statistics include average value; mean value, median value, standard * 
deviations, minimums and maximums of each requested network performance ' 
metric over the requested time period. For example, if a user requests throughput, . 
DNS lockup time, connect time, request time and response time for a specific ... 
URL oyer a given 48 hour period, the average value, mean value, median valua, r 
minimum value, maximum value and standard deviation for all the measurements 
taken of each of these metrics over the 48 hour time period is calculated Thus, if 
60 measurements of each metric are taken per hour, for each metric, the variance 
statistics average, mimirnirn, maximum and standard deviation of 2880 
measurements is calculated. In addition to these overall variance statistics, 
preferably, variance statistics for each of the requested metrics is also calculated 
for smaller increments of time. Preferably, but not necessarily, a single variance . 
statistic, the average, is calculated for this smaller increment of time. Fox example,'. . 
and without limitation, the average value of each requested metric over a 1 hour • 
period is preferably also calculated. 

£00491 providing to a user the mean, median, average, minimum, ma x i mum 

and standard deviation of each metric in the manner described above can 
advantageous ry allow the user to determine network performance over a period of 
time and determine whether a network service device is operating outside of 
tolerance at any given time during the relevant time period. Additionally, 
calculating and storing variance statistics for a given metric over predetermined 
time periods provides a baseline for performance of a network service over time, 
As such, a user can compare performance of the network service at any given time 
to the established baseline. As discussed in greater detail below, this can . 
advantageously allow a user to "filter out" systemic network problems, such as 
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f ■ P. ! 

networktrafflcloadi whlctinri^ vi 

... .particular time, in evaluating ihe perfoi^^ 

' | 700501 • A s noted above a system and methodin accordance with the'present ' ; k j . .■: *. 
• . invention preferably identifies Tor a user network performance parameters that are 

outside of acce|>tabteco;n^^ * . 

present invenu^m, in step 132, averages of subsets of measurements are compared ■...■>' 
.■ to a calculated standard deviation for .the metric as calculated from a larger group . • . f '. 
of measurements taken over>a longer time period. For example, and without • • ? • 

limitation^ if a user requests measurementsof a particular metric over a 48 hour • "j. 

period, and each requested metric is measured approximately once per minute, the . ■ . [ ■ 

• subsets of hourly averages (or other variance statistic), preferably calculated from . ■ 

60 measurements during the 48 hour, period can be compared to the same variance . . • • * 
statistic calculated for. the entire group of 2880 measurements taken over the 48 

• hour period. Preferably, regardless of whether a user requests data for a 48 hour m 
period or other length of time, the hourly average of each metric is compared to . " . 

the average value ofthe same metric over the entire requested time period, or, as - 7 - , . 
explained below; over another time period. It should be noted that herein, a subset . 
of measurements can include a single measurement. In such a case, the average for 
the subset exists and is considered to be the value of the sin^e measurement 
J r0Q51] . T he comparison that is made in step 132 preferably, though not 
necessarily, involves determining if the average (or other variance statistic) of a 
subset of measurements is within a pi^etennined number of standard deviations 
: from the average (or other variance statistic) of an overall group of measurements 
. from which the standard deviation was calculated. The subset or subsets of 
measurements can be part of the overall group of measurements but need not be. If ■ 
ihe variance statistic of a subset tif measurements is more rata a pr^ 
number of standard deviations away from the same . variance, statistic of an overall . . " 

group of measurement,' then the variance statistic ofthe subset of measurements is 
considered to be outside of acceptable tolei^ce cr, in other words, M 6ut of • ~ * ■ 
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control". "What- constitutea out of control perfb nuance, that is, how .many standard ■ 
deviations a variance statistic of a subset of measurements tgnst be away from the 
same variance statistic of a larger group of measurements, is preferably * 
configurable by a user. .*. 1 

. r00521 . C hart 1 below provides Sail example of the results of a potential . s 
measurement of end-to-end time (the time taken from the moment a user clicks on 
a link to aweb page to the instant the web page is fully downloaded and 
displayed) illustrating in cbntrol.and potentially out of control performance. In this, 
example, it can be assumed that the total measurement rime is over a 48 hour 
period and that end-to-end time pertaining to the relevant web site is being 
measured approximately once per minute. In Chart 1, the variance statistic that is . 
calculated and' compared is average value of the -end-to-end time. It could, 
however, be & mean, median, maximum, minimum or other such statistic of the 
end-to-end time. 





Chart 1 






EndrTo-End Time 




Hourly Average (Sec.) 


# Stil Dev. fiofii Collective 
Avg. 


Hour! 


4.5 


<1 


Honr2 


8.0 


>lbut<2 


Hour 3 


12.5 


>2 


Hour4 r 


42 


<1 


Collective Average 


4.0 




Standard Deviation 


2.0 





Ead-To-fio.il Tims 



(0053J 

T0054I The first four rows of the first column of chart 1 list the average of 

the 60 measurements taken during the firet four hours of the 48 hour time period. 
The fifth row of the first column lists the collective average of the 2880 
measurements taken of end-to-end time over the 48 hour period and the sixth row 
of the first column lists the standard deviation of this collective set of 
measurements. The second column of chart 1 displays the number of standard 
deviations each hourly average is away from the collective average. As shown, in 
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(-■ • f. l 

.. . hour.l,1he>verageerid-^^^ j b j^.^ry.y 

de\dation-away from the! collective average^ Acc^rdihgly, the eaidrto-end time .in • : \ \ \ r? • A 

. hour one Would likely be .considered within acceptable operating tolerance, or ixx :*, • j >. „ v} . Ji ,i ■ ; 
control. However, in hour 2 the holirty average was 8.0 seconds. This is more that " . ? r. > 

1 standard deviation: away from the. collective average but still less that 2 Standard, \ . 
^ deviatio^a^y from the collective average. Accordingly, the end-to-end time in .1*5 / - : 

. hour 2 might 'be considered but of control. In hour. 3* the hourly average is. 12.5. • ; ; 
seconds. This is grater than 2 standa^^ . .-v' " 

average and accordmglyvwom^ . - 

. hourly average is back % ytiAm'C6D^x6L at 4.2 seconds, which is less than one \ \ ' . " . 
. standard deviation" away from the coUective -average. 

J [00551 " W hether a metric is to or out of control is preferably determinable by * r 

the user. For example,, the user may determine that anything within 2 standard 
' deviations of a collective variance statistic is in control or that anything greater ■ ' , 

Any other scheme for detfe^^ 

.performance is out of control is also within the ambit of the present invention- For 
example, and without limitation, te"lM»b]i^oiL of whether a network ... ..... 

parameter is out of control or not could also be made using any other number of . 
stanc^d deviation . '*..,. ]. 

is greater than 1.5 standard deviations away from the, collective average or greater 
than 2.5 standard deviations away from the collective average, the parameter could 
' be considered out of control. . • • . * -I • 

fti6&>j . It is also contemplated to categorize performance in two or more ' . 
levek. For example, without limitation, any. measurement greater than 1 standard • ,; • 

. deviation from, the collective average but less that 2 standard deviations therefrom : * " V • . 
could be cgnsidered a first, or warjoing level. And, any measurement greater than 2 . : . • p.. " 
standard deviations away from a collective average would be considered a second, . t 
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ctfal^ level; Eacfr^ v-v 
.r\a&fi^^ , •.}... -.;v«:'.. ! 

:.: • j T0OS71 • I f an variance &atfefic of a subset of measurements is oufrof confrbl/ .v • j 

a problem cpuld.be ffidi^^ •■ ; 

. s. other comporieht.o£^ • .;• « . ~ 

'-. Accordingly, .in step 134, variance statistics which are out of control are 
highlighted to stand did from'other calculated and reported variance statistics for 
• '• tie user th^ request « r 
measurements are WgWighted depends Upon howthe information is to be reported • ... \ ~ 
to the user. Preferably, as explained in detail below, the requested measurements ... 

* and statistics are reported to the user in a tabular or chart format in a screen, 
display provided oh a user terminal. Using this reporting format, measurements 
that are out of control are preferably highlighted by displaying such measurement . 

.. ... in a different color than measurements that are in control Most preferably, 

• variance statistics that are within I to 2 standard deviations away from a collective \ 
. . variance statistic are Mgjbiighted in a first color and variance statistics that are % . ; 

greater than 2 standard deviations from a collective, variance statistics are ■ " 

highlighted hi a second color. In step 140, the results of the request made in step \ 
120 are reported back to 1^ iiser maldng the request ^ ~ 
preferably done by providing a screen dlspUy showing.the measuretn^ 
variance analysis resufb in tabular or chart fbrra^ 
. is also considered; however, that the. measurement and variance analysis results be . 
displayed in any other fbrmat such a graph showing averages (or other variance 
statistic) over time as. compared to standard deviations. It is also within the ambit . '• 

of the present invention, that the i^ortmg out step hicludes gen 
when one or mote metrics for one or more network services are out of control. To 
... initiate such an alarm, the method and system of the present invention could send 

and e-mail to,a predetermined address*; send a fax, or initiate a phone call. ■ 

>'''.' 
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rPOSyr. As discussed above with respect to Step 132, in one embodiment of 
method 100 it is preferable to compare hourly averages of measured network . 
' performance metrics -with.' collective averages of such metrics over a longer period 
. of time, siicfc as 48 homes, which includes, the hour feom which the hourly average - 
. was calculated- However, past network performance can vary s depending on time 
. of day, day or: week or even time of year. Specifically, for example, due to . 

different amounts of network traffic in the middle of a week day afternoon as 
• compared to early morning weekend times, a network service will likely display ■ . 
superior performance during the* early morning. weekend times. For example, due. 
. to varying network traffic, an average end-to-end time would likely be longer in 
the middle of a Friday aftemOon .than . early Saturday tnoming. 
f 00591 •.. ..A ccordingly, limits of acceptable network performance would likely 
be different for the two time periods*. Specifically, variance statistics, of network 
performance metrics for the high network traffic periods would reflect the feet that 
the network is uhdet heavy load; For example, DNS lookup tjmes, connect times, 
request times, packets lost and packets received may all be longer during a period 
. ofhigh network traffic than during a period of low network traffic. And, such 
longer time periods might not represent any type of network service malfunction, 
only that networkrtraffic is high. Thus, applying the limits determined for a 
consistently low network traffic period to a period ot consistently high network 
traffic period, or vice-versa* could produce misleading results. For example, if 
DNS lookup time is lower during periods of low network traffic, then a collective 
average applied to DNS lookup tiine during periods of high network traffic might 
be too. low and result iii false reports of out of control measurements. Conversely, 
including measurements for periods of high network traffic, in the collective 
average applied to periods of low network traffic could result in missing 
•measurements which might otherwise be considered out of control. Additionally,, 
mciuding periods of relatively high network traffic with periods pf relatively low 
network traffic could result in larger standard deviations. This could cause 
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variance statistics which might otherwise be* considered out of control not to be •. 
designated as such. ....... , . % 

j fo0601 A ccoiYHnglv. in step 132, it is.also wtthin the ambit of the presenti 

invention to compare performance at any particular time of interest to past 
: performance at a similar time.. For; example, network performance on* say, a 
Wednesday from 2:00 to 4:00 can ; be compared to network performance on any 
number of previous Wednesdays over the same time period as opposed to 
including other times and days Of the week in the comparison, such as a Sunday 
evening at 1 1 #0 p.m. when network traffic would likely be quite different 

. Preferably, if such a comparison is desired, it can be requested by a user in step 
120. In this way, past datafiom.a time frame similar to the time frame of interest 
can be used to perform the composite variance analysis in step 130. 

j fOOtfll • D etermining network perfonnance bv comparing network 
performance data with data collected at similar times can take into account 
systemic or environmental network conditions such as network traffic. Such a 
comparison allows systemic or environmental network conditions to be H filtered . 
out" so that operation of a network device or service can advantageously be 
isolated and accurately measured. 

r0Ofi21 ; F IG. 2 is a block diagram showing a preferred embodiment of a 
system 10 for measuring and reporting performance statistics for network 
operations in accordance with the present invention. As shown, system 10 includes 
at least one data collection agent (DGA) 20 which is preferably located on a 

distributed computer network 25 such as the jrrt grne^ could al so 

include, but is not limited to, a LAN (local area network), WAN (wide area 
network), MAN (metropolitan area network), VAN (value-added network), PAN , 
(personal area network), PON (passive optical network), YPN, enterprise-wide 
network, direct connection, active network; control network, an intranet, or any 
other suitable network. DCA 20 is preferably an automated server that carries out 
measurements on network 25, predetermined portions of or services provided on 
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network 25, of a device which is pait.of network 25. Services provided by network • 
25 can include, but are not limited to, URL objects such as web sites and 
streaming media objects. : ; : . _ . : s ;i 

| TOQ631 ' The type of rhejvaTrttnerii r^m'pyi mrf Ky TV 1 A 1(\ ^^0^^ up OI) fty* 

data to be : coUectedV For example, if DC A 20 is to collect data for a throughput 
. measurement of a given web .site, DC A 20 wHl connect to the given web site and . 
measure the throughput for a predetermined amount of time. Preferably, as 
discussed above, thfe .measurement will be Seated on aperiodic basis for a given ' 
duration of time. Measurement of URL object and str eamin g media object 
parameters,. such as those listed in the Background section, by DCA's is well 
understood by those skilled in the art 

f0064I Preferably. DCA 20 is pre-configured with information concerning 
what measurements are, to. be taken, at what times and on whjch network services. . 
J Such pre-configurarion of a DCA is ^Mjg^^ogct;by those skilled In the art For „ 
web page or steaming media measurements, the configuration preferably contains 
the "URL. or location of the streaming media object for which a given test is to be 
performed. The configuration also contains the network performance metric to be 
tested and the frequency of the test to be performed (for example, once each 
minute, preferably at substantially evenly spaced intervals). 
fft06Sl . P referably, DCA 20 is configured to take measurements of at least 
each of the following URL or streaming media service rnetrics once each minute 
On a continuous basis: tirroughput, DNS lookup time, connect time, request time, . 
server response time, connection time, socket teardown time download time, 
and/or end to end time. With respect to streaming media services, such a metrics 
can include, but are not limited to: packets lost, packets received, bytes received, • 
jitter, percent packets lost, DNS lookup time, buffer time average bandwidlh, 
and/or first stat time. However, which metrics are measured and when they are 
measured can be determined in advance by a user. 
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data for One or more URLs or service metrics named in the subscription. DCA 20< ■. *.: A 
can be a Windows NT servers, UNIX servers or any other type of server. : 
Configuration and unplmentation of a IX3A ^ch as DCA 20 is.weU understood ; " ■ - 
bytfios^skmedintfteart • • -, 

I r00671 j_After b^ta^ c^ec^ b^ 
ingest module 30 which transforms the data into an appropriate format for storage; . 
as is well understood by those skilled in the ait Preferably, DCA 20 forwards data .1 . • • »; 
to data ingest module 30 on a regular, pfedeteimined basis. Information 

| concerning when this data jae to be fcy^ 
configuration information. Data ingest module 30 then forwards the data to 
perfonnance data repository 40 which is preferably, but not necessarily, placed at 
.q. network location apart from DCA 20. Performance data repository 40 can be any 
type of database or a plurality of databases capable of storing network 
peiformance metric data collected by DCA 20. It is also considered, however, that 
perfonnance data repository could be ajjy faciUty for storing text strings. 
Preferably,' however, performance data repository 40 supports Structured Query 
LanguAjge (SQL). 

j r00681 System 10 also includes processing software 50 which performs 

statistical analyses on network performance metric data collected by DCA 20 and 
stored in performance data repository 40. Processing software 50 preferably runs 
on a processing server 55 and will be discussed in greater detail below. System 10 
also includes web browser 70 which provides a user (not shown) with access to . 
processing server 55 and processing software 50. Data collected by DCA 20 
remains iri database 40 until a user request 72 is initiated by the user of web 
browser 70. Web browser 70 is preferably a standard computer terminal including - 
a display monitor hut can be any device, such as a PDA or cellular telephone, 
capable of cornmunicating with processing server 55. 
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| lfig§a User request72 P referably ineludes Mmtifying information for the *\ . .." : 
UKL or. streaming media service, netwo* device, or other portion of n^ 
, the user wishes to anal 

.and/or date range over winch the user wishes to retrieve data. Additionally, as 
discussed above, if the user wishes to compare data from, the guested time and ..'.:„■ 
. daterange to a similar, but different tim^^^ . . 

include the similar time and date raiige to which the requested time and dale range 
is t6 be compared User request 72 is received by data access module 52. ■ •• * ] . 
; Preferably, data access module 52 tonstnicts a queiy for da^ i^ository 40 to 
' retrieve the raw data from data repository 40 necessary to generate the requested 
composite variable statistics. ....... 

| f0Q70l As discussed above,' DCA '20 preferably measures a wide range of 

metrics of predetermined network services on a continuous basis. Arid, most • \[ . 

preferably, such measurements are taken approximately once^ each minute. Also as . 
discussed above, all these measurements are. stored in data repository 40. Further, ... : 
data repository 40 preferably retains all the data provided to it by data ingest 
module 30 for a predetermined period, such as 3 monAs. As such, a request for 
data constructed by data access module 50 can preferably retrieve data from any 
time frame within the predetermined data retention time period for any metric 
measured for a monitored network device or service. Also, because data repository 
40 preferably stores each measurement made and measurements are preferably m \ , 

made in appromnatety one minute increments, data^ prefoahl£re^^ 
minute increments. Construction of a query to retrieve data from a data repository, 
such as data repository is well understood by those skilled in the art. " 
mill After retrieving the required raw data from data rftpndtmy 4Q, Hat* ■ 

access module forwards the raw data to recordset processing module 54. Recordset: 
processing module 54 preferably completes requested composite variance analysis , 
on raw measurement data collected by DCA 20. Additionally, recordset processing . • , 
module 54 preferably constructs components of a display of the calculated . / . 
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statistics. Tie display is to-be used to present the calculated statistics to the • 
network analyst on wet browser. 7<X Preferably, recordset processing module 54. . 
. constructs components of a display using a computer markup language such as 
HTML- 

f007ai, ;_Jlecordset processing module 54 includes compute statistics module 

54a, compare .module 54b and biiild.compdneri.ts;module 54c, Compute statistics .-. 
module 54a of rewrdsetprocessing inodule 54 preferably completes the composite 
variance analysis of the raw data provided by data access module 52. Compute, 
statistics module 54a accepts data sets for each metric mcluded.in.the query results • 
. and calculates statistics oyer predetermined time periods for each of the data sets. 
Preferably, compute statistics module 54a can be configured to. calculate, any 
statistics for a data set over any time range. More preferably, hqwever, compute • 
statistics module 54a calculates the following statistics over the associated time ;• 
periods; mean value, median value, average value* standard deviation, minimum 
value and maximum value for the data set associated with each metric for the 
entire time period requested (e,j^^ 

associated with each metric. Calculation of scich statistics from sets of data is well - 
understood by those skilled in the art, 

100731 A fter computing the requested statistics from the provided raw data, 

compute statistics module 54a preferably provides the calculated statistics to 
compare module 54b, In an embodiment of the present invention in which a 
system and method identifies for a user network performance parameters that are 
outside of acceptable control limits, compare module 54b compares given variance . 
statistics calculated by compute statistics module 54a to determine if a 
predetermined relationship exists. between the given variance statistics. Preferably,, 
though not necessarily, compare module 54b compares averages, means, medians, 
minlmums and/or maximums of subsets of measurements to a» average, mean, 
median, minimum and/or maximum vahie calculated from a larger group of 
measurements taken over a longer time period. Preferably, compute statistics ; 
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module S4a determines if tfh& variance statistic associated with the subset of . . .• :.w; 

. measurements (e. r avargga [ : triegT^ ;mg^^ jgd^jgm Jg^OT maa^ u^ j^ggg; ■ . . \ — f: 
a predetermined relation^ associatea* v : ;i 

with the larger group of measurements. If apredetfertnined relationship does (or, .. \ m , / ■ , 

alteriiarively, does;iiot):ex^'betwee^ 6ompai^ vkiance statistics, compare . - t . . 

module 54b identities the variance statistic associated with the subset of ; .... / .I 

: measurements for which the predetermined relationship did .(or did not) exist 

| 100741 . F or example, and wifhout Jicnitarion, if a user requests measurements . ; 
of a particular metric over a 48 hour period, and each requested metric is n^^ 
approximately once per minute, the subsets of hourly avenges (including 60 
measurements) during the 48 hour period can be compared to the standard 
deviation 1 calculated for the entire group of 2880 measurements taken over the 48 
hour period. Preferably, regardless of ^etherauserxeqpests d^fora48hour • 
period or other length >of time, the hourly average of each metric is compared to \ 
the- average value of the same metric over the entire requested time period, or, as ; ■ < . 

'explained below, over .another time period As noted above, a subset of. . , > . - 

measurements can include a single measurement. In such a case, the average for 
the subset exists and is considered to be the value of the single measurement 

| t007 ^1 . M ore preferably, though not necessarily, compare module 54b 
determines if the average of a subset of measurements is within a predetermined 
number of standard deviations from the average of an overall group of . 
measurements from which the standard deviation was calculated. The subset or . 
subsets of measurements can be part of the overall group of measurements but • 
need not be. If the average of a subset of measurements is more than a 
predetermined number, of standard deviations away, from the average of an overall 
group of measurements,, then the average of "the slubset of measurements is . " 

. considered to be "out of control" . Compare module 54b can then flag or otherwise 
identify averages of the subset of measurements that are out of control. What ( 
constitutes out of control performance, that is, how many standard deviations an -■ . 
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m average of a subset .of measirremenfs must be away from the. average of a larger 
. . 'group of measurements, ii inferably configurable by a uservPetfon&ing the- : \ 

above descxribed. comparisons betweejgt .(calculated variance statistics, and 
. identifying variance statistics which do notmeet certain.coiKtitoa5> is well 

• uiideistocMibyAoseskinedintheait... 

j f00761 Preferablv^aflBT performing the above described comparisons and; . 

identifying, measurements tbat are out of control, compare module 54b forwards : 
this infoTination to. build components rttadule 54e. Build components module 54c : 
preferably constructs components of a display of the calculated variance Statistics > 
Preferably, build components module 54c accomplishes this using a computer 
markup language such at hypertext markup language (HTML). After building 
components of the measurement and statistics display, build components module . 
54c preferably forwards the components to page rendering module 56 which 

. interprets the output of build components module 54c and preferably displays the • 
measurements and statistics display on web browser 70. As noted above, build 
components module 54b preferably constructs an HTML document. Accordingly, . 
page rendering module 56 preferably renders the measurement and statistics 
display using HTML. 

r0077T F IG. 3 is a screen print showing a preferred embodiment of a 
measurements and statistics screen display 210 rendered by page rendering 
module 56 in accordance, with the present invention: Performance analysis line 
212. of screen display 210 provides identification information for the service or . 
network component for which calculated statistic are displayed. Specifically, in 
FIG. 3, the displayed statistics relate to a web site having the inU^sjhown ,infl<j w 

- " ■ : • • 

mm . .Bene ath performance analysis line 212, display chart 214_ displays. 
measurement and statistics relevant to the displayed XJEL. Date column 216 
displays the date on which corresponding measurements were made. Hour column \ . 
218 displays the time over which the corresponding averages were made. Jh the ■ 
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. example shown by display .c^^ % .• ; " v.- 

corresponding, averages were calculated is one hour. However, it is considered (hat 
" * hour .column 218 could also display any other timfcperiddcr^ : * 

measurements were made or. displayed averages or other statistic were, calculated. * r 
i ■ j [OOggl . . Statistics column 220 displays statistics calculated from the ;. " 

. measurements talcen by DC A 20 relevant to a user selected metric. The selected 
metric is displayed at the top of statistics column 220. M the example shown in . • .*. •> s . \ : . 
. . ■ FIG. 3, the selected metric i£ end-to^end time* However, any measured metric can ; 
preferably be selected. Beneath the column label is .displayed the statistics which 
have been calculated for the relevant metric. In FIG. 3, statistics column 220 is 
preferably divided into an additional^ columns displaying far each hour • 
. measurements were taken, an average value, minimum value, maximum value and . 
standard deviation. •,*».. 
| 100801 . . As discussed above, in a preferred embodiment each measurement 
of each metric is taken approximately once each minute. Accordingly, the 1 , 

averages, minirmims, maximums and standard deviations, displayed in statistics " , . • 

column 220 represent statistics calculated from approximately 60 measurements. . 
The hour and dale associated with each statistic displayed is preferably provided in . 
the same row as the statistic. Thus, for example screen display 210 shown in FIG. , [ t - 
3, the average end-to-end time for the web site listed on Jun. 10, 2001 at 1600 .... 
hours was 4.35 seconds. The minimum end-to-end time in that hour was 1.59 . . 1 

seconds, the maximum was 1 5:24 seconds and the standard deviation for the 
.measurements taken over the hour was 3144. It is also cor^idered to display . . • . 

\:- additional or fewer statistics in statistics cqlumn-2?0.. < - .-..= * 

j ISggfl Display chart 214 also preferably includes errors column 222 which. . ' " 

t . .. : displays tfcenu^ ' ; • 

the corresponding dateiand hour. As discussed above, DGA 20 preferably records . . 
. , three types of errors: .access errors,- service errors and content errors. Errors \ 
:.; column 222 is preferably divided into three additional columns to display each of > . ; ? 
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these types of- errors for the corresponding date and hour in which the errors !*"... 
occurred. For example, there were 13 content errors with respect to the relevant : ? 

web site duringthe firsthonrof Jutl 11- 2001; , ;•:.« -v. 

100821 ' D isplay Chart 214 also preferably in rJndftc: tnfn I m^aqntiMYi^titc ! :^ 

• cohunn 224 which displays the total number of measurements ta^^ .;.«■<....'. ;•; 

» corresponding time, period (shown in" hour column 21 8> For example, while 60 • 'J , . ■ 

measurements were, taken during most hours on lun. 10, 2001, at 21Q0 hours on /, 
fbiat day, only 59 measurements were taken. • . . . : > " " •/ v 

T00831 D isplay chart-214 also preferably includes performance details .'..'V s 4 . 
column 226, which displays one statistic for the measurements made during the 
corresponding time period (shown jba hour Cttlun^ 

displayed is the average, value of the metric listed at the top of the listing column ■■•.'■*. 
during the corresponding: time period In display chart 

most sub-column- of performance details column 226 displays the average, value of ■ 
the eod-to-end tinie during the corresponding date and time. Thus, on Jun. 10, 
2001, the average of the 60 measurements taken duinig 1900 hours of end-to-end . 
time of the measured web site was 4.68 seconds. 
J [00841 . P referably, f^rfbnaance details column 226 is divided into as many . 
additional columns as metrics for which measurements were taken during the 
relevant time period. For example, as shown in FIG. 4, which is second view of 
display chart 214 showing the entire performance details column 226, sub- , ' : . ~ 
columns for the following measurements are included in performance details 
column 226: end-to-end time, throughput, DNS lookup, time, connect time, request . \ 
time, response time, teardown time, and dowrdoadtime.lt is also considered to ■ 
include fewer or additional (if DC A 20 took measurements of additional metrics V r 
with respect to the particular object being me^ured) type^ of melrics in 

performance details column 226. ■ . t : 

| rd085l . A s discussed above, a system and method in accordance with the '~\ 
present invention can preferably use. history of performance data statistics to alert . • . ■; . ■. 
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a user afeout network p^rfoiraance that is outside acceptable tolerance or control :.•.-•••> 

limits. HG. 5 is ascreen display showing* portion of .display, chart 214 of screen. ; . .;. ....... 

. display 2UMb exemplary display chart 214, processing software. 50 highlights a .} •; _ ; . 

measurement statistic ina first icotor, a$ discoid above, if the measui^ent ...... 

- rtatistic is mort that l;5.standai« deviations, bm . . ,. , , ... r . 

away from a collective average and highlights a measurement statistic in.* second .:. . ' . 

colorifthe measurement statistic' is ^ .... 

coUertve average. As ... \ 

ia the- "Avg" cdlucan and "Std Dev" column) the collective average fox end-to-end. * - : ; , 
time is 4.99 seconds and the collective standard deviation is 4.07. However, for \ 
, hour 1000 on JuilH, 2001, the average end-to-end time was B.58.secpnda. This . . 

is more than 2 collective standard deviations .(4.07) away from the collective ... 
average end-to-end time (4.99 seconds). Accordingly, the average end-to-end time ■ ■ 
at hour 1000 has.been highli^ited. . .... 

| Purther, ihe tbrougbputfor hour 1000 on Jubl 1 1, 2001 (26.90 

KB/Sec) has been highlighted in a different color. Accordingly, while the user is I. .. 

notinfonned on display chart 214 of whatthe collective average or standard . 
deviation for throughput is, the user is alerted that the througj^ 
greater than 1 standard deviation but less that 2 standard deviations away from the 
collective average; ' 

| f0087] As shown in FIGS. 3-5 and discussed above, statistics screen display 

210 preferably displays hourly averages of a wide range of metrics related to a 
network obj ect in a relatively compact tabular format and highlights measurements 
which reflect that a network service may be but of coiitroL As such, a system axui 
method in accordance with the present invention consolidates and presents a 
relatively large ^ . . , 

advantageously allows a user viewing statistics screen display 210 to quickly 
complete a relatively thorough assessment the performance^ a selected network . .. 
object and detetmine whether any corrective actions are necessary with respect to .'•[>'.•' 
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peifon»anceLofa:n^ork5aWce^ • ;. . 

f008g) ; ■ r A s noted abo^itis also within the ambit o f the present inveitfpft to : ;• V , / . 
display results' 6f performance measurements and statistical calculations and \ : / • ; . . . ..... 

c»mpariso^manyd^ . „ . 

j I00S21 — L^The foregoing descriptions of spwific embodiments of the pr^ent 
.myratioiih^^ V 

not Intended , to bo exhaustive oy. to limit the invention -to the precise Jfonns. , ■ \'.' } 
disctosed,aMitsho 

: possible in light of the above teaching. The embodiments were chosen and ... * 
described teorde* to b . 
application, to 1hereby..enabb oft . 
and various embodiments with various modifications as are suited to the particular .* 
use contemplated. Marry other variations are also to be considered within the • 
scope, of the pre^ irrVentiorL For instance, a system and method in accordance . ~ 
with tte.present invention can measure, analyze and report network performance, , 
metrics other than those enumerated in the Background section including network ; 
performance statistics that are not associated with a URL or streaming media 
object. For example, a system and method Jn accordance with the present 
invention could measure, analyze and report network performance metrics • ■ 

associated with network haidwaredevi " ... 

network; 
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Application No. 10/743,732 
of Stephen CARLE Y 

Amendments to the Drawings 

Please add the attached two new drawings (Figures 3 A and 6). 
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